{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6ae0145b-00e8-4e81-810b-6e90854e4a91",
   "metadata": {},
   "source": [
    "# LangChain RAG with Local Models\n",
    "\n",
    "This is based on Pixegami's tutorial. ([original repo](https://github.com/pixegami/rag-tutorial-v2/))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbdd4d05-ddac-4ccf-b71d-650ad8f92d0b",
   "metadata": {},
   "source": [
    "## Download data and folder setup\n",
    "\n",
    "On the **Docker host side**, run the following to set up the `jetson-containers`' `/data` directory.\n",
    "\n",
    "```\n",
    "cd jetson-containers\n",
    "mkdir -p data/documents/L4T-README\n",
    "cp /media/jetson/L4T-README/*.txt data/documents/L4T-README/\n",
    "```\n",
    "\n",
    "This in turn creates the mounted volume `/data/documents/L4T-README` inside the container.<br> \n",
    "Your directory structure should look like this:\n",
    "\n",
    "```\n",
    "└── ./data/documents/L4T-README\n",
    "    ├── INDEX.txt\n",
    "    ├── README-usb-dev-mode.txt\n",
    "    ├── README-vnc.txt\n",
    "    └── README-wifi.txt\n",
    "```\n",
    "\n",
    "You can check this with running a following bash command in the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "973053ec-7c55-4126-a8cc-a66ef65a5aa4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/data/documents/L4T-README:\n",
      "total 24\n",
      "-rw-rw-r-- 1 1000 1000  1104 May  6 22:42 INDEX.txt\n",
      "-rw-rw-r-- 1 1000 1000 11126 May  6 22:42 README-usb-dev-mode.txt\n",
      "-rw-rw-r-- 1 1000 1000  3590 May  6 22:42 README-vnc.txt\n",
      "-rw-rw-r-- 1 1000 1000  1940 May  6 22:42 README-wifi.txt\n"
     ]
    }
   ],
   "source": [
    "!ls -Rl /data/documents/L4T-README"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36e251a0-24f1-484d-a278-d513e4f4cd36",
   "metadata": {},
   "source": [
    "## Loading The Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3a24edf3-1751-4f11-a1d0-0ca1921e1014",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import DirectoryLoader\n",
    "\n",
    "DATA_PATH = '/data/documents/L4T-README'\n",
    "\n",
    "def load_documents():\n",
    "    document_loader = DirectoryLoader(DATA_PATH, glob=\"*.txt\")\n",
    "    return document_loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "922717b9-f225-463d-8b87-9a37a8b85435",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /root/nltk_data...\n",
      "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "page_content='1======================================================================\\n\\nINDEX\\n\\nLinux for Tegra\\n\\nIndex of README files\\n\\n=======================================================================\\n\\nThis directory contains various documentation for Linux for Tegra:\\n\\nREADME\\n\\nusb\\n\\ndev\\n\\nmode.txt:\\n\\nDescribes how Linux for Tegra configures the Tegra USB flashing port in USB device mode, and the features that are exposed to a connected host PC, which include network and virtual serial port connectivity.\\n\\nREADME\\n\\nwifi.txt:\\n\\nDescribes how to connect Jetson to a WiFi network using command-line tools. This is useful when connecting to Jetson using ssh over the USB device mode network connection.\\n\\nREADME\\n\\nvnc.txt:\\n\\nDescribes how to configure Jetson to run a VNC server using command-line tools. This configuration can be performed using ssh over the USB device mode network connection. This allows use of graphical applications without attaching an HDMI display, keyboard, and mouse to Jetson.' metadata={'source': '/data/documents/L4T-README/INDEX.txt'}\n"
     ]
    }
   ],
   "source": [
    "documents = load_documents()\n",
    "print(len(documents))\n",
    "print(documents[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de6ce522-e9f9-4521-afb3-788df1e37ef1",
   "metadata": {},
   "source": [
    "## Split The Documents "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0ec0ab31-8796-45fe-b445-4575698576de",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "from langchain.schema.document import Document\n",
    "\n",
    "def split_documents(documents: list[Document]):\n",
    "    text_splitter = RecursiveCharacterTextSplitter(\n",
    "        chunk_size=800,\n",
    "        chunk_overlap=50,\n",
    "        length_function=len,\n",
    "        is_separator_regex=False,\n",
    "    )\n",
    "    return text_splitter.split_documents(documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "274f4d4e-af46-4596-857a-aa478a1563d8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27\n",
      "page_content='1======================================================================\\n\\nINDEX\\n\\nLinux for Tegra\\n\\nIndex of README files\\n\\n=======================================================================\\n\\nThis directory contains various documentation for Linux for Tegra:\\n\\nREADME\\n\\nusb\\n\\ndev\\n\\nmode.txt:\\n\\nDescribes how Linux for Tegra configures the Tegra USB flashing port in USB device mode, and the features that are exposed to a connected host PC, which include network and virtual serial port connectivity.\\n\\nREADME\\n\\nwifi.txt:\\n\\nDescribes how to connect Jetson to a WiFi network using command-line tools. This is useful when connecting to Jetson using ssh over the USB device mode network connection.\\n\\nREADME\\n\\nvnc.txt:' metadata={'source': '/data/documents/L4T-README/INDEX.txt'}\n"
     ]
    }
   ],
   "source": [
    "chunks = split_documents(documents)\n",
    "print(len(chunks))\n",
    "print(chunks[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11bdce8c-f27d-4aee-a785-46b14f218804",
   "metadata": {},
   "source": [
    "## Embedding Models\n",
    "\n",
    "There are couple of options for the embedding model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9154e5f8-aa7e-4212-98c5-c439b1247410",
   "metadata": {},
   "source": [
    "### Running **Local** Embedding Model\n",
    "\n",
    "#### Option 1: Open embedding model running locally\n",
    "\n",
    "Besides the LLM, you can run the embedding model locally on Jetson as well.\n",
    "\n",
    "Example embedding models available on `ollama` are listed on an [Ollama blog article](https://ollama.com/blog/embedding-models).\n",
    "\n",
    "| Model | Parameter Size | Link |\n",
    "| ----- | --------------:| ---- |\n",
    "|`mxbai-embed-large`|334M|[link](https://ollama.com/library/mxbai-embed-large)|\n",
    "|`snowflake-arctic-embed`|334M|[link](https://ollama.com/library/snowflake-arctic-embed)|\n",
    "|`nomic-embed-text`|137M|[link](https://ollama.com/library/nomic-embed-text)|\n",
    "|`all-minilm`|23M|[link](https://ollama.com/library/all-minilm)|\n",
    "\n",
    "Here, we try `mxbai-embed-large`, which had proved to generate good enough embeddings for our sample documents."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f33da5e-3122-49a3-b837-852950abe486",
   "metadata": {},
   "source": [
    "First, check you have already downloaded the embedding model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7d57c73e-2556-406e-be8b-feea695ec81e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                    \tID          \tSIZE  \tMODIFIED       \n",
      "mxbai-embed-large:latest\t468836162de7\t669 MB\t45 minutes ago\t\n",
      "llama3:70b              \tbe39eb53a197\t39 GB \t16 hours ago  \t\n",
      "llama3:latest           \ta6990ed6be41\t4.7 GB\t16 hours ago  \t\n",
      "nomic-embed-text:latest \t0a109f422b47\t274 MB\t23 hours ago  \t\n",
      "llama2:latest           \t78e26419b446\t3.8 GB\t10 days ago   \t\n",
      "mistral:latest          \t61e88e884507\t4.1 GB\t2 weeks ago   \t\n",
      "llama2:70b              \te7f6c06ffef4\t38 GB \t2 weeks ago   \t\n",
      "llama2:13b              \td475bf4c50bc\t7.4 GB\t2 weeks ago   \t\n"
     ]
    }
   ],
   "source": [
    "!ollama list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2209208e-7ba6-46b8-8954-222591677916",
   "metadata": {},
   "source": [
    "If not, use `ollama pull` command to pull the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b9c94e8-9a4a-40c6-a032-08dfdda40a7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ollama pull nomic-embed-text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "70dfc81e-a824-4db2-b6b2-111c676d2b98",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_community.embeddings.ollama import OllamaEmbeddings\n",
    "\n",
    "def get_embedding_model():\n",
    "    embedding_model = OllamaEmbeddings(model=\"mxbai-embed-large\")\n",
    "    return embedding_model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61fb24d6-9974-49ba-8c7f-99656fe46a7f",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Using Cloud Embedding Model\n",
    "\n",
    "Some people believe cloud hosted embedding modles perform more accurately.\n",
    "\n",
    "#### Option 2: OpenAI Embedding model\n",
    "\n",
    "[API Reference: `OpenAIEbmeddings`](https://api.python.langchain.com/en/latest/embeddings/langchain_openai.embeddings.base.OpenAIEmbeddings.html)\n",
    "\n",
    "> **Remember to put YOUR own OpenAI API key in the following cell.** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5863023e-787c-43b8-8994-c6a960927c14",
   "metadata": {},
   "outputs": [],
   "source": [
    "OPENAI_API_KEY = \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48fbb2a1-1526-4e01-9125-e9b8b63c3bd2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_openai import OpenAIEmbeddings\n",
    "\n",
    "def get_embedding_model():\n",
    "    embedding_model = OpenAIEmbeddings(\n",
    "        model = \"text-embedding-3-large\",\n",
    "        openai_api_key = OPENAI_API_KEY\n",
    "    )\n",
    "    return embedding_model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ce566bf-c6bd-4396-99d7-5bd670a3b8d9",
   "metadata": {},
   "source": [
    "#### Option 3: AWS Bedrock Embedding model\n",
    "\n",
    "[API Reference: `langchain_community.embeddings.bedrock`](https://api.python.langchain.com/en/latest/llms/langchain_community.llms.bedrock.Bedrock.html)\n",
    "\n",
    "> **Note, you need to set up AWS profile separately.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ff88cb0-5475-4e14-968a-e3f92f729c04",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_community.embeddings.bedrock import BedrockEmbeddings\n",
    "\n",
    "def get_embedding_model():\n",
    "    embedding_model = BedrockEmbeddings(\n",
    "        credentials_profile_name=\"default\", region_name=\"us-east-1\"\n",
    "    )\n",
    "    return embedding_model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4886a89e-9721-4294-ab57-0c8c807e6386",
   "metadata": {},
   "source": [
    "## Creating The Vector Store\n",
    "\n",
    "We are going to create the vector store with embeddings and save it in a directory as files.<br>\n",
    "Here, the directory is defined to be \"**chromadb**\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c839a7a0-da54-4d41-9e46-b900288b6b60",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "CHROMA_PATH = \"chromadb\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a0da60e-c833-4e36-aef0-098534c394fe",
   "metadata": {},
   "source": [
    "Remove the directory if it has previously been created and populated.\n",
    "\n",
    "> If you are re-running with different embedding model, removing the persisted directory may not be enough.<br>The work-around would be to restart the Python kernel.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "342409dc-4726-4192-8eff-904848724c7a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%bash -s \"$CHROMA_PATH\"\n",
    "rm -rf $1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eabafded-e353-497d-9083-22c2af44f83a",
   "metadata": {},
   "source": [
    "### Vector store to be created with embedding model specified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "140646fa-9214-42d6-973c-ede42e1ffd3d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.vectorstores.chroma import Chroma\n",
    "\n",
    "def add_to_chroma(chunks: list[Document]):\n",
    "    vectorstore = Chroma(\n",
    "        persist_directory=CHROMA_PATH, \n",
    "        embedding_function=get_embedding_model()\n",
    "    )\n",
    "    vectorstore.add_documents(chunks)\n",
    "    vectorstore.persist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2cf77a7b-9065-4cf0-af54-fdebf0d26264",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:119: LangChainDeprecationWarning: Since Chroma 0.4.x the manual persistence method is no longer supported as docs are automatically persisted.\n",
      "  warn_deprecated(\n"
     ]
    }
   ],
   "source": [
    "add_to_chroma(chunks)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b58fe9ab-2b19-40c4-8e65-9aa791264b2b",
   "metadata": {},
   "source": [
    "Let's check what files are saved and the size of each file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c6d87c22-c199-4c37-9d89-8cd5e1c43721",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.0K\t./chromadb/451ecaa3-56ba-4076-afb8-793a024dce9e/length.bin\n",
      "4.0K\t./chromadb/451ecaa3-56ba-4076-afb8-793a024dce9e/header.bin\n",
      "4.1M\t./chromadb/451ecaa3-56ba-4076-afb8-793a024dce9e/data_level0.bin\n",
      "0\t./chromadb/451ecaa3-56ba-4076-afb8-793a024dce9e/link_lists.bin\n",
      "4.1M\t./chromadb/451ecaa3-56ba-4076-afb8-793a024dce9e\n",
      "404K\t./chromadb/chroma.sqlite3\n",
      "4.5M\t./chromadb\n"
     ]
    }
   ],
   "source": [
    "%%bash -s \"$CHROMA_PATH\"\n",
    "du -ah ./$1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40432bc9-56fc-4e5e-9215-45357e68a528",
   "metadata": {},
   "source": [
    "## Running RAG Query Locally"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b9c45d9-3ffe-4ec6-a26d-977c0773ef4f",
   "metadata": {},
   "source": [
    "Below defines the template for the prompt to eventually sent to our LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9168af39-7e5d-4214-b5c4-98e606eac77c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "PROMPT_TEMPLATE = \"\"\"\n",
    "Answer the question based only on the following context:\n",
    "{context}\n",
    "\n",
    "---\n",
    "Answer the question based on the above context: {question}\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1d44eaf-78f1-4041-a495-38d120cd3287",
   "metadata": {},
   "source": [
    "The actual question is supplied as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "06efab23-a279-4f25-9751-f3b75721f8ee",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "query_text=\"What IPv4 address Jetson device gets assigned when connected to a PC with a USB cable? \\\n",
    "    And what file to edit in order to change the IP address to be assigned to Jetson itself in USB device mode? \\\n",
    "    Plesae state which section you find the answer for each question.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "384822c9-346c-4a84-9bdf-3919dbbdeca3",
   "metadata": {},
   "source": [
    "### Load vector store from persisted files with embedding model specified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6a3b4c0c-4606-41d2-996b-c1629b0cd778",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "vectorstore = Chroma(\n",
    "    persist_directory=CHROMA_PATH, \n",
    "    embedding_function=get_embedding_model()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1414545-0a99-4f5c-a2c6-8d5d1ddcbdd0",
   "metadata": {},
   "source": [
    "### Search the vector store for retrieving relevant context\n",
    "\n",
    "Top 5 relevant chunks are retrieved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3ff520e3-da35-4051-a05b-7d9ade16aa8d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "results = vectorstore.similarity_search_with_score(query_text, k=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "fc4dd700-6c1b-45b9-aa81-2306d89f9e4a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.prompts import ChatPromptTemplate\n",
    "\n",
    "context_text = \"\\n\\n---\\n\\n\".join([doc.page_content for doc, _score in results])\n",
    "prompt_template = ChatPromptTemplate.from_template(PROMPT_TEMPLATE)\n",
    "prompt = prompt_template.format(context=context_text, question=query_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc0d1b1a-4cc7-4759-abe8-f2c2787d6dfc",
   "metadata": {},
   "source": [
    "Final `prompt` is generated with context filled.\n",
    "\n",
    "If you decide to print out the generated prompt, run the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "76e650c5-219e-4413-989d-935a3a631770",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Human: \n",
      "Answer the question based only on the following context:\n",
      "Linux for Tegra assigns a static IPv4 address of 192.168.55.1 to Jetson, and runs a DHCP server to automatically assign an IPv4 address of 192.168.55.100 to your host machine. This provides point-to-point connectivity. If a Jetson device experiences very high CPU or disk IO load, this DHCP server may fail to respond in a timely manner to requests from the host machine. This may cause IPv4 connections to drop. If this problem occurs, configure your host machine to use a static IPv4 address of 192.168.55.100 with netmask 255.255.255.0 and no gateway or DNS servers.\n",
      "\n",
      "---\n",
      "\n",
      "If you connect multiple Jetson devices to the same host machine, each Jetson device uses the same IPv4 address. This prevents IPv4-based communication with all but one Jetson device, since your host operating system determines which Jetson device it communicates with. To solve this, edit the Jetson-based script that sets up the network and assign a unique network address to each Jetson device. IPv6-based communication works with any number of Jetson devices without modifying the aforementioned script.\n",
      "\n",
      "Once the Ethernet device is configured, use SSH to connect to the Jetson device.\n",
      "\n",
      "SSH is natively available on Linux and Macintosh operating systems. Connect using IPv6 with the following command:\n",
      "\n",
      "ssh nvidia@fe80::1%usb0\n",
      "\n",
      "Connect using IPv4 with the following command:\n",
      "\n",
      "---\n",
      "\n",
      "Linux for Tegra assigns a link-local IPv6 address of fe80::1 to the USB Ethernet device(s). You should not have to configure your host machine in order to use an IPv6 link-local address. If you connect multiple Jetson devices to the same host machine, each Jetson device uses the same IPv6 link-local address. You may still connect to any connected Jetson device, however, since use of IPv6 link-local addresses also requires specifying which network interface to connect over, and there is a unique interface for each Jetson.\n",
      "\n",
      "---\n",
      "\n",
      "Ethernet on Mac ---------------------------------------------------------------------- An NCM USB Ethernet device is created, and the required driver is automatically activated. Since the Mac operating system does not support the RNDIS device, the device is not activated by default.\n",
      "\n",
      "Changing the IPv4 Address ---------------------------------------------------------------------- Edit /opt/nvidia/l4t-usb-device-mode/nv-l4t-usb-device-mode-config.sh on Jetson to change the IPv4 network parameters. The following variables must be changed, and must maintain consistent values: - net_ip - net_mask - net_net - net_dhcp_start - net_dhcp_end\n",
      "\n",
      "---\n",
      "\n",
      "Connect using IPv4 with the following command:\n",
      "\n",
      "ssh nvidia@192.168.55.1\n",
      "\n",
      "On Windows, use the PuTTY application.\n",
      "\n",
      "Ethernet on Linux ---------------------------------------------------------------------- Two USB Ethernet devices are created and the required drivers are automatically activated. These devices are typically named usb0 and usb1, depending on how your Linux distribution is configured, and what other hardware is attached to the system. To avoid the potential for interference between the two USB Ethernet devices, it is best, but not strictly required, to use your host's network configuration tool (e.g. Network Manager), to disable one of the two USB Ethernet interfaces implemented by the Jetson device.\n",
      "\n",
      "View available Ethernet devices with the following command:\n",
      "\n",
      "/sbin/ifconfig\n",
      "\n",
      "---\n",
      "Answer the question based on the above context: What IPv4 address Jetson device gets assigned when connected to a PC with a USB cable?     And what file to edit in order to change the IP address to be assigned to Jetson itself in USB device mode?     Plesae state which section you find the answer for each question.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae09240a-fc5e-4ff4-ad28-3860f86526fa",
   "metadata": {},
   "source": [
    "### Define LLM\n",
    "\n",
    "Define the local LLM using `Ollama` to be invoked with the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e4bff3e0-e38e-44ce-92e9-28a82c9aed57",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_community.llms.ollama import Ollama\n",
    "\n",
    "model = Ollama(model=\"llama3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c2f99d9-4c87-4c26-9ca0-36e9ceaed438",
   "metadata": {
    "tags": []
   },
   "source": [
    "If you have not downloaded `llama3` model and the above cell failed, run the following cell and come back to execute the above cell again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36920eb4-c80a-497f-a8bf-a6ed240b393e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!ollama pull llama3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03d47f1a-2fc8-4797-9977-0401a445cfde",
   "metadata": {},
   "source": [
    "#### Alternative: Using OpenAI LLM\n",
    "\n",
    "In case you wanted to try OpenAI LLM, you can run the following cell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7690356-2ce9-43bd-8bcb-51dcee0ba79f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_openai import OpenAI\n",
    "\n",
    "model = OpenAI(\n",
    "    model=\"gpt-3.5-turbo-instruct\",\n",
    "    openai_api_key = OPENAI_API_KEY\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc0bc50e-631e-4357-9f06-728df550c45b",
   "metadata": {},
   "source": [
    "#### Running the LLM \n",
    "\n",
    "The following cell runs the Llama3 model on Ollama with the prompt.\n",
    "\n",
    "> If you open a Terminal on the side (you can dock a pane on the side on JupyterLab) and runs `jtop`, you can check the GPU and other resource usage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7dceda73-3665-4ec2-bed0-20ab40ce68e2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "response_text = model.invoke(prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae7d03f3-2d9b-416d-a598-b7fd88b8daf8",
   "metadata": {},
   "source": [
    "Let's check the LLM output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8ade3c38-d92b-47a0-9d6e-b6ba2d77d928",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Based on the provided context:\n",
       "\n",
       "**What IPv4 address Jetson device gets assigned when connected to a PC with a USB cable?**\n",
       "\n",
       "According to Section \"Linux for Tegra\", Linux for Tegra assigns a static IPv4 address of 192.168.55.1 to Jetson, and runs a DHCP server to automatically assign an IPv4 address of 192.168.55.100 to the host machine.\n",
       "\n",
       "**What file to edit in order to change the IP address to be assigned to Jetson itself in USB device mode?**\n",
       "\n",
       "According to Section \"Changing the IPv4 Address\", you need to edit `/opt/nvidia/l4t-usb-device-mode/nv-l4t-usb-device-mode-config.sh` on Jetson to change the IPv4 network parameters."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, Markdown, Latex\n",
    "display(Markdown(response_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a772677-e1dd-4272-85a3-324f9b9ac7b5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
